Reconstructing individual monophonic instruments from musical mixtures using scene completion

Monaural sound source separation is the process of separating sound sources from a single channel mixture. In mixtures of pitched musical instruments, the problem of overlapping harmonics poses a significant challenge to source separation and reconstruction. One standard method to resolve overlapped harmonics is based on the assumption that harmonics of the same source have correlated amplitude envelopes: common amplitude modulation (CAM). Based on CAM, overlapped harmonics are approximated using the amplitude envelope from the nonoverlapped harmonics of the same note. CAM assumes nonoverlapped harmonics from the same noteare available and have similar amplitude envelopes to the overlapped harmonics. This is not always the case. A technique is proposed for harmonic temporal envelope estimation based on the idea of scene completion. The system learns the harmonic envelope for each instruments notes from the nonoverlapped harmonics of other notes played by that instrument, wherever they occur in the recording. This model is used to reconstruct the overlapped harmonic envelopes for obstructed harmonics. This allows reconstruction of completely overlapped notes, yet does not require predetermined instrument models. Experiments show the proposed algorithm performs better than an existing system based on CAM when the harmonics of pitched instrument are strongly overlapped.